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ABSTRACT 
The Lorenz, 1963 model is a simple model that exhibits features such as nonlinear chaotic behavior and the existence of regimes 
similar to the actual climate system. It can be used for studying the predictability of climate. The Lorenz model has been forced by 
an external forcing in order to imitate anthropogenic forcing in the climate system. It also has relevance to monsoon predictability. 
Inclusion of forcing and moving average has been shown to increase the predictability of the Lorenz model. Statistical methods, 
such as the Artificial Neural Network (ANN) are often used for making predictions. In this study, we explore how the predictability, 
using ANN, of Lorenz data is affected by external forcing and moving average. It is hoped that such a study can throw some light on 
the effect of anthropogenic forcing on climate predictability. Three ANN architectures have been used for this purpose, namely — 
Feed Forward Back Propagation Neural Networks (MLP), Distributed Time Delay Neural Network and Nonlinear Auto Regressive 
Neural Network. None of the architectures revealed any change in predictability due to external forcing but the moving average 
results to increase in the overall predictability of the system. It is also shown that the predictability of the Lorenz data using ANN is 


Hades 7 


ARTICLE 


independent of the local dynamics of the system. This study suggests that because of the low computational cost of ANN and its 


non-dependence on local dynamics of the system, ANN may be more appropriate for short term climate prediction in regions of 
high dynamical error growth rate. 
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1. INTRODUCTION 

The Lorenz-63 model is considered as a basic model that shows the chaotic behaviour similar to the climate system (Lorenz, 
1963).There are various studies, which have shown that the forcing as well as time averaging results in increase in predictability 
(Shukla 1981, Palmer 1993, Dwivedi et.al. 2007, Li Ai-Bing et.al. 2012). The study by Mittal et. al. 2003 has shown that the forced 
Lorenz model is relevant to the monsoon predictability by considering the forcing function to represent the rate of advancement of 
monsoon trough towards the Bay of Bengal/foothills of Himalayas. 

The dynamical method like Bred vector growth rate to study the local predictability of Lorenz attractor has shown the existence 
of regions of distinct predictability (Evans et. al. 2004). Similar types of regions of dynamical error growth and decay have been also 
evident by using eigenvalues of symmetric Jacobian matrix (Mittal et. al. 2015). These regions of dynamical error growth and decay 
suggest that there are regions, in the attractor, which are highly predictable while the other is less predictable. In the study by Mittal 
et. al. 2015, it has been observed that the prediction error by using statistical methods do not depend on the state of the system 
because the statistical method predicts the state of the system by using values of past observables. The statistical methods have 
predicted the time series of the Lorenz model with the similar accuracy for both regions of error growth and decay. 

The statistical method, like Artificial Neural Network (ANN), is widely being used for time series prediction of physical and natural 
processes. The intent of the prediction is to determine the future behavior of the system on the basis of past behavior of the system 
(Oullette & Wood, 1998). The ANN has been proven to be a useful technique for prediction of complex time series because of its 
ability to determine law governing the evolution of the system from regularities in the past (Gershenfeld and Weigend 1993, De 
Gooijer and Hyndman 2006, Hatalis et. al. 2014). 

Generally, an artificial neural network is considered to model the way in which the brain performs a particular task or function of 
interest. The adaptivity and generalization are the essential capability of artificial neural network (Haykin, 2002). The ability to predict 
Lorenz attractor by using ANN has been investigated in many research studies (Pasini et. al. 2005, Sanjay et.al. 2005, Pasini 2008, 
Woolley 2010). 

In this study, two basic approaches have been used to study the effect of anthropogenic forcing on the climate prediction. In the 
first approach, the Lorenz model is forced with the forcing parameters described in Mittal et. al. (2003, 2005) and then its 
predictability is assessed. In the second approach, the moving average is applied to the obtained time series of forced Lorenz model 
in order to assess effects of the moving average on its predictability. 

The rest of this paper is structured as follows. In section 2, we cover the basic concept like methodology and ANN architectures 
used. Sections 3 will cover the results of this study. Finally, in section 4 conclusion of this study will be given. 


2. BASIC CONCEPTS 


The Lorenz-63 model is governed by three equations given as follows: 


dx 
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The forcing terms F,, Fy and F, are added to the above Lorenz equations (Palmer 1994) in order to get Forced Lorenz model 


equations and it is represented by following system of equations: 
dx 
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where, values of o = 10, B = 8/3, p = 28. The Lorenz model exhibits chaotic behavior for these values of o, B and p. 


The prandtl number o represents the ratio of the fluid viscosity to its thermal conductivity, p is the Rayleigh number and & is 
geometric number. According to studies by Mittal et. al. (2003, 2005) and Dwivedi et. al. 2007, the forced Lorenz model provides 
simplified mathematical analysis for forcing parameter values to be F,= aF, Fy=-F, Fz=0, where F = 1.5. The equation 2 is integrated 
upto 105,000 time steps using variable-order Adams—Bashforth—Moulton PECE (predictor-evaluate—corrector—evaluate) solver. In this 
study, the initial condition (1, 1, 1) and integration time step Dt = 0.01 is used. The 5000 initial transient points were discarded and 
the remaining points were assumed to belong to the Lorenz attractor. 


2.1. Jacobian and Bred Vector Growth Rate 
In this work we have used two dynamical methods for finding regions of error growth and decay as suggested in Mittal et.al. 2015. 
In the first method, regions of dynamical error growth and decay have been calculated by using Jacobian. In principal, the error 
growth rate lies between the minimum and maximum eigenvalues of symmetric Jacobian matrix J; If B is considered to be a region 
in which all the eigenvalues are negative, then all the infinitesimal perturbations will decrease irrespective of its orientation as long 
as the trajectory in B region. On the other hand, If R is considered to be a region in which all the eigenvalues are positive, then all 
the infinitesimal perturbations will increase irrespective of its orientation as long as the trajectory in R region. 
In the second method, Bred vector growth rate has been used to calculate the region of error growth and decay (Pasini and Pelino 
2005, Pasini 2008). A periodically rescaled difference between two model runs, starting from slightly different initial condition is 
known as bred vector. Evan et. al. 2004 had predicted transition of the regime in the Lorenz model by means of Bred vector growth 
rate. 

The calculation of blue (B) and red (R) regions from symmetric Jacobian matrix J; and B' and R' region have been done by using a 
similar method described in Mittal et.al. 2015 and for brevity it is not discussed here. In figure 1, the B and B’ (specified by blue 
color) are the error decay regions, while R and R’ (specified by red color) are the regions of error growth. 
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Figure 1 
The left panel shows the forced Lorenz attractor plotted in blue (red), the maximum eigenvalues of J; is negative (positive). The right 
panel shows the forced Lorenz attractor plotted in blue (red), the bred vector growth rate is negative (positive). 


2.2. Artificial Neural Networks 
In this study three ANN Architecture has been chosen. All these Neural Networks have their own mechanism and have been used in 
many studies for nonlinear time series predictions. 


2.2.1 Multilayer Perceptron (MLP) 

The MLP is also known as feed forward back propagation neural network. It can be used for nonlinear mapping from of input data 
to output data. The first layer is called the input layer, the last layer is called the output layer and the layers in between are hidden 
layers. Figure 2 shows the MLP architecture used in this study. 
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Figure 2 MLP architecture 


2.2.2 Distributed Time Delay Neural Network (DTDNN) 

Dynamic networks are used to make a neural network responsive to the temporal structure of information-bearing signals. Neural 
network is made dynamic by providing memory to it. The short-term memory to the neural network is given by using time delay. 
The time delay can be implemented into ANN at the synaptic level inside the network or in the input layer (Haykin, 2002). In DTDNN 
(shown in figure 3) implicit influence of time is distributed throughout the network. Such networks are constructed as the spatio- 
temporal neuron model on the basis of the multiple input neuronal filters. 
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Figure 3 DTDNN architecture 
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Figure 4 NARNET architecture 


2.2.3 Nonlinear Auto Regressive Neural Network 
Another type of network we use in our analysis is a nonlinear autoregressive neural network, or NARNET (shown in figure 4). It is a 
dynamic recurrent network based on Elman recurrent network architecture. In this type of network neurons depend not only on 
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external inputs, but also on their own lagged values. NAR (nonlinear autoregressive) neural networks can be trained to predict a 


time series from that series past values. Elman network builds “memory” in the evolution of neurons. This type of network is similar 
to moving-average (MA) process that is often used in financial time-series analysis. In the MA process, dependent variable y is a 
function of independent variables x as well as current and lagged values of a random shock €. 


3. RESULTS 

In this study all ANN architectures are designed with one input layer, one hidden layer and one output layer. To overcome 
limitations of a linear dimensionality reduction ANN is designed with the sigmoidal activation function in the hidden layer and the 
linear activation function in the output layer. The Levenberg-Marquardt training algorithm (Bishop 1995), which minimizes the sum 
of squares error, has been chosen for training purpose of ANN. 

The data was serially divided into two sets, namely Training set and Test set respectively in the ratio of 80% and 20%. The 
optimization of the model complexity for given training data set is done by using cross-validation techniques. To apply cross- 
validation technique, the training set was further divided into two random sets, Estimation set and Validation set in ratio 87.5% and 
12.5%. The training is stopped by using early stopping criteria (Bishop 1995); when the error in the validation set start to increase as 
the network starts to over-fit. The network with the smallest error with respect to validation set is selected. The above mentioned 
approach is called hold-out method. Finally, the performance of Test set is assessed to confirm that there is no over fitting in the 
selected ANN. The number of hidden layer neurons is one of the important parameter while designing a neural network. In this 
study 20 neurons have been used in the hidden layer of all three ANN architectures. All ANN architectures have been trained several 
times in order to get best network performance. 

In the present work, Prediction error was measured by taking the Euclidean distance between Lorenz data (target data) and ANN 
output. It has been used as a criterion for performance evaluation of the ANN model. The prediction errors for the training 
(estimation set) and test set were found similar in all cases. This insures that no over fitting occurred and thus, all the models have 
generalization capability. In Table 1 prediction errors of the forced Lorenz model by using ANN architectures is listed. 


Table 1 Prediction Errors for Forced Lorenz model 


Regions of MLP DTDNN NARNET 
error Median+ Median + Median+ 
Growth/Decay Mean + Std iqr/2 Mean + Std iqr/2 Mean + Std iqr/2 
Blue(B) 0.010 + 0.011 | 0.009 + 0.004 [0.010 + 0.006 |0.009+ 0.005 |0.010 + 0.011 /0.010 + 0.004 
Red(R) 0.010 + 0.011 |0.009 + 0.004 |0.010 + 0.005 |0.009+ 0.005 /0.010 + 0.012 |0.010 + 0.004 
Blue'(B') 0.010 + 0.011 |0.009 + 0.004 |0.010 + 0.005 |0.009 + 0.005 /0.010 + 0.011 |0.010 + 0.004 
Red'(R’) 0.010 + 0.012 |0.009 + 0.004 |0.010 + 0.005 |0.009+0.005 |0.010 + 0.012 /0.010 + 0.004 
Table 2 Prediction error with moving average window size = 40 

Regions of MLP DTDNN NARNET 
error Median + Median + Median + 
Growth/Decay Mean + Std iqr/2 Mean + Std iqr/2 Mean + Std iqr/2 
Blue(B) 0.008 + 0.007 | 0.007 + 0.003 /0.008 + 0.007 |0.007 + 0.002 | 0.009 + 0.011 | 0.009 + 0.002 
Red(R) 0.008 + 0.006 |0.007 + 0.002 /0.008 + 0.005 |0.007 + 0.002 |0.009 + 0.010 | 0.009 + 0.002 
Blue'(B’) 0.008 + 0.006 | 0.007 + 0.002 /0.008 + 0.006 |0.007 + 0.002 | 0.009 + 0.011 | 0.009 + 0.002 
Red'(R’) 0.008 + 0.006 | 0.007 + 0.002 /0.008 + 0.009 |0.007 + 0.002 | 0.009 + 0.011 | 0.009 + 0.002 
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On comparing ANN prediction results obtained in Mittal et.al. 2015 for the unforced Lorenz model with the results obtained for 
the forced Lorenz model (Table 1 in this study), it can be observed that the prediction errors are almost same in both cases. There is 
no significant decrease in prediction errors by forcing the Lorenz model. Although small increase in the number of data points is 
observed by using dynamical methods in the regions of dynamical error decay B and B’. The effect of moving average on the 
predictability of time series obtained by forced Lorenz model is listed in the Table 2, and 3 for moving average window length 40 
and 120 steps respectively. It can be observed that the prediction errors for all ANN architectures are decreasing. It means that the 


predictability increases by applying moving average on the Lorenz model. 


Table 3 Prediction error with moving average window size = 120 


Regions of MLP DTDNN NARNET 
error Median + Median + Median + 
Growth/Decay Mean + Std iqr/2 Mean + Std iqr/2 Mean + Std iqr/2 
Blue(B) 0.006 + 0.006 |0.005 + 0.002 |0.006 + 0.006 |0.005 + 0.002 /0.006 + 0.007 /|0.005 + 0.002 
Red(R) 0.006 + 0.006 |0.005 + 0.002 |0.006 + 0.005 |0.005 + 0.002 /0.006 + 0.005 /|0.005 + 0.002 
Blue’(B’) 0.006 + 0.006 |0.005 + 0.003 |0.006 + 0.005 |0.005 + 0.002 /0.006 + 0.005 /|0.005 + 0.002 
Red'(R’) 0.006 + 0.006 |0.005 + 0.002 |0.006 + 0.005 |0.005 + 0.002 /0.006 + 0.005 /|0.005 + 0.002 


The correlation coefficient between ANN prediction errors and Max eigenvalues of J; for all the above cases were found to be in 
the range of -0.04 to -0.01 which is significant at the 99.9% level. The correlation coefficient between ANN prediction errors and the 
bred vector growth rate for all the above cases were found to be in the range of -0.03 to -0.01 which is significant at the 99.9% level. 
The plot between ANN prediction errors and the bred vector growth rate is shown in figure 5 for all the three ANN architectures. It is 
clear from the figure that ANN prediction errors do not depend on the region of error growth and decay, for all the three ANN 
architectures because the spread of errors is almost similar in both cases (bred vector growth and decay). 
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Figure 5 
ANN Prediction errors and bred vector growth rate (left panel for MLP, middle panel for DTDNN and right panel for NARNET) 


4. CONCLUSION 

The ANN prediction results listed in table 1, from which it can be concluded that none of the architectures revealed any significant 
change in predictability due to external forcing. Dwivedi et.al. 2007 have shown by using Lyapunov exponent as a measure of the 
predictability, that the effect of forcing on the predictability of attractor is small. This might be considered, as one of the possible 
reason that is why none of the ANN architecture is able to capture the increase in predictability due to forcing. Thus, finer tuning of 
parameters in ANN architectures is needed in case to capture the effect of forcing on the predictability of Lorenz model. It can be 
concluded from the table 2 and 3 that the moving average results to increase in the overall predictability of the system since 
prediction error decreases with the increase in length of the moving average window. Similar result of increase in the predictability 
of the forced Lorenz system by applying moving averages has been also found, with the help of Laypunov exponent and Shanon 
entropy, in a study by Dwivedi et.al. 2007. They have shown that with increase in length of moving average window size the largest 
Laypunov exponent and Shanon entropy decreases. Thus, the predictability of system increases. The attractor splits into more 
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regimes and occupies a small region due to forcing and increase in moving average window size. An attractor with small size has 


less saturation error and it results in increase in its predictability (Kennel et. al. 1994). Although, the forcing and moving average 
causes increase in predictability, but the results listed in table 1, 2 and 3 also reveals the non dependence of prediction by statistical 
methods on the region of dynamical error growth and decay since prediction errors in the error growth regions (R and R’) are almost 
similar to the prediction error in error decay regions (B and B’). Finally, the results of this study suggest that because of the low 
computational cost of ANN and its non-dependence on local dynamics of the system, ANN may be more appropriate for short term 
climate prediction in regions of high dynamical error growth rate. 
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